1. Field of the Invention
This invention relates to augmented reality (AR) systems that provide instructions to guide a user's retrieval and use of parts in the performance of a task and more specifically to an AR system that identifies and tracks the parts throughout a natural scene and visually cues the user to identify and locate parts both inside and outside the user's current field-of-view (FOV) to execute each instruction.
2. Description of the Related Art
Augmented Reality (AR) refers to the computer generation of 2D or 3D graphics or other media such that they are rendered and overlaid on and registered with the displayed images of surrounding objects in the environment. Applying AR to tasks such as maintenance or assembly involving parts and equipment makes it possible for users to be trained for those tasks, and actively assisted during their performance, without ever needing to refer to separate paper or technical manuals. Incorporating instruction and assistance directly within the task domain, and directly referencing the parts and equipment at which the user is looking, eliminates the need for personnel to continually switch their focus of attention between the tasks and the manuals.
Many AR systems in use today utilize a video camera and computer vision algorithms to find and identify predetermined visible patterns in the scene. Such patterns might be specially designed fiducial markers (e.g. a 2D bar code) or they might simply be pre-stored images or photographs that exhibit unique structure. Having found a visible pattern in the scene, the AR system can then infer the 6-dimensional pose (X/Y/Z position and roll/pitch/yaw angular orientation) of the camera with respect to the visible pattern. If the visible pattern is known to be fixed to a specific location in the world, the AR system may calculate the position of the camera within the world. The AR system can render graphics that overlay the image of the world captured by the camera in such a way that those graphics would point to specific physical locations in the scene. Alternatively, the AR system can render graphics that have a pre-determined three-dimensional relationship to a moveable object in the scene, provided that one or more of the visible patterns in the scene has a known and fixed spatial relationship to the moveable object in question.
As the camera moves through the scene, the image of the visible pattern changes according to the camera's new position or orientation. Computer vision algorithms can re-compute the new pose of the camera with respect to the visible pattern. Such computation may be simpler and faster if the visible pattern has only moved slightly compared to the last time such a computation was performed. However, if the camera moves to such an extent that the visible pattern partially or fully departs the visible scene (or the visible pattern becomes occluded by other objects in the scene), it is no longer possible to compute the pose of the camera by analysis of that visible pattern. Under such circumstances, many AR systems lose knowledge of their location with respect to the world or lose knowledge of the location of the object to which the visible pattern is attached. Such knowledge may be re-acquired anew only when the visible pattern returns to the scene in a fully visible manner. Therefore, during the time that the visible pattern is not in the field of view, an AR system may lose the ability to correctly render graphics that refer to physical locations or objects in the scene. Alternately, the AR system can attempt to acquire a different visual pattern in the currently visible scene and restart the process anew.